Image Stitching: Exploring Practices, Software and Performance

The merging or stitching of separately captured portions (tiles) of an object into a single unified digital image is becoming increasingly popular in the cultural heritage community. Maps, negatives, tapestries, and paintings that were once too onerous to digitize faithfully because of their physical size are now included in digital collections. These can be digitally sewn together from component images with several post-processing solutions. In some cases large robotic systems are accomplishing these tasks. When such stitched images are viewed without a known reference image the stitching performance can appear quite remarkable. What and where are the hidden flaws in these stitched objects? Are certain content types more prone to stitching errors than others? Are there analytical tools to detect stitching errors or are visual assessments sufficient? What operational guidelines and software options offer the best stitching solutions? And, as is often the case, do these tools cater to other imaging sectors with quite different sensitivities than those of cultural heritage institutes. We explore these questions and offer an assessment of best current thinking on the pros and cons of different digital stitching solutions and guidelines on how to make them perform well. Introduction As the need to faithfully digitize large flat objects (larger than A0) with sufficient resolution has increased, so too have the solutions for doing so. A classic and simple solution has been to physically scale common linear array scanner hardware to match expected object sizes. The concept is simple: move a large flat object past an imaging detector head in a precisely timed manner to capture single lines of image data that are then sequentially combined to yield a finished image file of a two dimensional object. While not often thought of as image stitching, this approach is indeed the most basic and accurate form of stitching: line after line of image data is combined by the scanner’s hardware environment to yield a larger ‘stitched’ image in two dimensions. Some linear array scanners will even move the linear array side to side over a wide platen area and stitch the sideway image components together for greater areal coverage. Combining these image components is accomplished within the scanner’s image processing pipeline and is invisible to the user. These are examples of integrated imaging systems, where high level knowledge about the scanner’s position, performance and movements is available beforehand. This approach is used to achieve very high stitching accuracy. In this sense, the image stitching process has near perfect ‘vision.’ Such methods, and intelligence, are used on a grand scale today using two-dimensional step-and-repeat robotic systems employing rapid capture devices. These are typically used to digitize large vertically mounted artwork (typically paintings or large murals) [1] that is difficult to move, and must be scanned insitu. These tend to be highly constrained closed systems, whose imaging performance parameters (geometric distortion and vignetting), are calibrated beforehand and compensated for in postcapture processing. However, the total ownership costs (e.g., training, equipment, software, and maintenance) can be high, and outside the budgets of most institutions. A more common, economical, approach uses capture devices, like digital SLR cameras or linear array scanners, with limited fields of view to capture multiple sub-images that together cover an entire object of interest. For this community these objects tend to be large maps, newspapers, tapestries, and even large format negatives where higher sampling rates (i.e. dpi, ppi) are required. These environments are distinguished from stitching of large artwork in several ways. Less high-level calibration information. Generally speaking, well-characterized lens data, or resources to reliably measure them, are simply not available for most users. Having this lens correction information available with software to exploit it helps greatly in accurate stitching reconstruction. Lack of these data requires blind processing, based on general assumptions. This can result in poor results. Demanding productivity requirements. Large amounts of time to manually edit images to remove stitching artifacts are just not acceptable for most imaging environments. While computational execution time to stitch images together is acceptable, manual intervention must be held to a minimum. Challenging content types and usage. Stitching errors near abrupt high contrast features with strong rectilinear visual cues, especially in the midst of large monotonous image regions, can be quite objectionable. Especially for maps, accurate stitching is critical because of the spatial geometry requirements. The stitched image is an object of information, not just a picture meant for casual viewing. We concentrate our study and exploration in this paper on this digitization environment since it is the predominant case for cultural heritage imaging institutes. Underlying Software Operations And now I see with eye serene, the very pulse of the machine. Henry Wordsworth, She was a Phantom of Delight The basic operations underlying an image stitching operation include; 1. Identifying approximate relative location of the various component (tile) images. 2. Identifying corresponding image features in each of the overlapping regions. 3. Selection of stitching boundaries, margins, for each set of overlapping regions. 4. Correction for camera distortion or perspective differences, based on image intensity differences and locations of corresponding features. This step will often involve resampling of the image information in regions far from the stitching boundaries, in order to deliver a continuousappearing composite image. 5. Merging (combining) of the image (pixel-value) data at and near to the stitching boundaries. As stated above, a good approach for image processing, but especially operations which involve object identification and content interpretation, is to supply available a priori information whenever practical. For our applications this will take the form of; • Relative location and orientation of component images, alleviating step 1. Furthermore, if these are related by simple translation (overlapping tiles at regular intervals), this reduces the loss of detail in the final image (step 4). • Known or reduced camera distortion and perspective differences. This helps with step 4. • Reduced image intensity differences (due to illumination variation) between component images. This reduces the severity of the processing in step 5. Figure 1, from Ref. 2 shows several of the steps in modern image stitching software. The example is shown for the stitching of a neighboring image to a ‘target’ image. The second step shows the identification of an overlap region, based on several corresponding features. As seen in the third step, a candidate stitching path is identified by searching the two images for a low-variation (called ‘minimum error’) path. This is done so as to reduce the visibility of the stitching boundary in the final composite image. The two sets of image data are merged (combined) at and near the boundary. In this case, some features in the target image on the left are replaced by more uniform regions in the second image. Note that, for this example, both input component images are assumed corrected (resampled) for any spatial distortion and differences in sampling and rotation. Figure 1: Outline of steps used in current image stitching software (from Ref. 2) We now describe a simple experiment aided at investigation the likely imaging performance when image stitching is working well, with a highly textured scene. As an example of a complex scene, consider Fig. 2, from a digital camera. This input scene was then cropped into four overlapping tile images, as indicated by the rectangles superimposed on the scene. These four sub-images where then stitched together using Adobe Photoshop software, which worked well. While displaying the stitched image in the ordinary way on a computer monitor, the differences between original and stitched image were not visible. Figure 3 shows a cropped section of the stitched image. Figure 2: Scene used in tiling investigation, showing the four tiles formed from cropping the original. Figure 3: Section of the above image after stitching To visualize the image differences introduced by the image stitching operation, an error, or difference image is shown in Fig. 4 for the same region as shown in Fig. 3. We show the pixel-by-pixel difference (stitched – original) for the green color-record. Note that the difference image does contain information about the scene content of the original image. This is due to the, in this case minor, spatial processing of the tile image data to correct any changed in image sampling or rotation. In our case there should have been no changes in either of these parameters. However, parameters for these operations are estimated from the image data by the software. Small changes in the estimated sampling interval and rotation parameters (part of any robust object-oriented image processing) lead to such spatial processing. As we see, the influence of these differences extends far from the stitching path. Sometimes this type of image difference can reduce the apparent sharpness of the stitched image content, although the effect was not severe in this case. Figure 4: Difference image for the region shown in Fig. 3, for the green color-